Running experiments

Old school:

1. Read a bit & come up with an idea
2. Prepare and run experiment
3. Prepare data & explore different analytic approaches
4. Significant result → Write paper

Experimenter degrees of freedom, incentives, issues

• What if… → Garden of forking paths (Rubin 2017)

• p-hacking, hypothesizing after the results are known (i.e. HARKing)

• The need for significance and novelty

• False positives research

Psychology Replication crisis

Is there a crisis? Why?

Improving Replicability

• Improve methods: increase n, better measures and manipulations, improve design, piloting

• Reduce temptations: p-hacking, hypothesizing after the results are known (i.e. HARKing), selective reporting → preregistration, internal replications

• Openness: transparency of the research process, sharing methods, materials, procedures, and data

Nosek et al. (2022)

Registered reports (RRs)

RRs were conceived to alter the incentives for authors and journals away from producing novel, positive, clean findings and towards conducting and publishing rigorous research on important questions. Soderberg et al. (2021)

How RRs work

• Write introduction, method, … before collecting data!
• Send to journal for review
• Revise and resubmit (improve before collecting data)
• Once you get In principle acceptance, collect human data, run analysis, write up, and send for a final review

RRs advantages

• More open, preregistered, reproducible by default

• It does not matter if p value is < 0.05

• Less incentives for p-hacking

• More trustworthy results

Registered reports are great

But isn’t this a bit…

Our path towards RRs

Background

We (CSCN; ~5-10 PI’s) used different technologies to develop experiments: Psychopy, Qualtrics, Limesurvey, jsPsych, etc.

Each of these has advantages and disadvantages.

Mostly, pragmatic aspects guided the decision: lab history and resources, coding experience, type of experiment (EEG/behavioral, lab/online), …

Issues

• Each protocol started almost from scratch. A single pre-existing task would define the technology used.

• Multiple implementations of the same tasks in different technologies, not always exact replicas, not always easy to find.

• Some would work in certain computers, other did not.

• Output data wildly different, data preparation a hard task and error prone.

Issues Survey

Experiment issues

• Errors in experiment logic
• Errors in items coding
• Data not what we expected
• Data structure makes data preparation hard
• Match between hypotheses and data not clear
• Variables or questions not used in the analysis/paper

Resources issues: projects as islands

• Hours wasted re-programming tasks
• Thousands of € ‘invested’ in licenses (e.g. Qualtrics)
• Piloting protocols as a part-time job for Research Assistants
• Hours wasted re-doing data preparation (each software has its own output format)

Reproducibility issues

• Anyone knows why this 2012 paradigm/data analysis is not running?
• Location and organization of projects
• Data preparation/analyses so ugly, sharing them is hard (let me clean up this a bit before sharing it with you)
• Idiosyncratic analyses, some of which require licensed closed software (SPSS, Matlab,…)

Our wish list

Our wish list

• Open source software based on standard technologies
• Reusable tasks (my project feeds future projects)
• Based on a mature project or technologies
• As many ‘automagic’ things as possible
• Easy to create and analyze paradigms
• Balancing participants
• Online/offline

jsPsychR

Goal

A big catalog of tasks in jsPsychMaker. Each task runs with jsPsychMonkeys to create virtual participants, and have a script in jsPsychHelpeR to automate data preparation (re-coding, reversing items, calculating dimensions, etc.).

The final goal is to help you have the data preparation and analysis ready before collecting any real data, reducing errors in the protocols, and making the move towards registered reports easier.

The team

Current developers:

• Gorka Navarrete, Herman Valencia

Initial idea and development:

• Gorka Navarrete, Nicolas Sanchez-Fuenzalida, Nicolas Alarcón, Alejandro Cofre, Herman Valencia

Discussions, ideas, testing:

• Esteban Hurtado, Alvaro Rivera, Juan Pablo Morales, …

jsPsychR tools

jsPsychMaker

Create New Tasks

Creating new tasks is as simple as:

1. Copy example tasks to your computer

jsPsychMaker::copy_example_tasks(
  destination_folder = "~/Downloads/ExampleTasks"
  )

2. Copy paste items, adapt csv/excel file

3. Create protocol (see next slide)

Create protocol

Create a protocol with three existing tasks plus the adapted Example tasks:

jsPsychMaker::create_protocol(
  canonical_tasks = c("AIM", "EAR", "IRI"),
  folder_tasks = "~/Downloads/ExampleTasks/",
  folder_output = "~/Downloads/protocol999",
  launch_browser = FALSE
)

jsPsychR tools

jsPsychMonkeys

Features jsPsychMonkeys

• Fully open source (R, docker, selenium)
• Online and offline
• Sequentially and in parallel
• Get pictures of each screen
• Store logs to make debugging easier
• Watch the monkeys as they work for you
• Random pauses or refreshing to simulate human behavior
• Set random seed to make the monkey’s behavior consistent

Release monkeys

Release a single Monkey and take a look:

jsPsychMonkeys::release_the_monkeys(
  uid = 11,
  initial_wait = 0,
  wait_retry = 0,
  local_folder_tasks = "~/Downloads/protocol999/",
  open_VNC = TRUE
)

Release 10 Monkeys in parallel:

jsPsychMonkeys::release_the_monkeys(
  uid = 1:10,
  sequential_parallel = "parallel",
  number_of_cores = 10,
  local_folder_tasks = "~/Downloads/protocol999/",
  open_VNC = FALSE
)

jsPsychR tools

jsPsychHelpeR

jsPsychHelpeR

Create project for data preparation:

jsPsychHelpeR::run_initial_setup(
  pid = 999,
  data_location = "~/Downloads/protocol999/.data/",
  folder = "~/Downloads/jsPsychR999"
  )

Create a task correction script:

jsPsychHelpeR::create_new_task("MultiChoice")

Visualize and run data preparation:

targets::tar_visnetwork(targets_only = TRUE, label = "time")

targets::tar_make()

Challenge: everything in 3 minutes?

Create protocol, simulate participants and prepare data…

# Full process
rstudioapi::navigateToFile("R/script-full-process.R")

Survey Experiment Issues

Survey results

Let’s try to download the data, process it and show a report with the results:

Plan A: run Experiment Issues project

rstudioapi::openProject("../jsPsychHelpeR-ExperimentIssues/jsPsychHelpeR-ExperimentIssues.Rproj", newSession = TRUE)

Plan B: If something fails, we always have the monkeys!

browseURL("../jsPsychHelpeR-ExperimentIssues/outputs/reports/report_analysis_monkeys.html")

Limitations

• Very easy to create new scales, and simple tasks, but complex experimental tasks require javascript and HTML knowledge (although there are a good number of examples available)

• Data preparation for new experimental tasks requires expertise in R (simple surveys not so much)

• Analysis reports require some R knowledge (simple templates available)

• Requires access to a server for online tasks

• Only behavioral tasks

Too many things, too little time

• Create templates for most common experimental designs
  
• Templates for data preparation and analysis of common experimental designs
  
• More tasks, translations, tests, …
  
• So far, development based in our needs (and possibilities)
  
• Upgrade to latest jsPsych when available
  
• Improve, clean, share…
  
• Publish jsPsychR paper

Help

• Javascript programmers

• R programmers

• Task creators

• Testers

Back to Registered reports

Can jsPsych really help? (1/2)

• With jsPsychR, protocols are standardized with (mostly) clean code, open source, and based on web standards

• Data preparation is 90% automatic, standardized, and beautiful

• Less errors in protocols and in data preparation

• Time from idea to protocol much lower

• When errors are found and fixed, old protocols can benefit from the corrections, old results can be checked, …

Can jsPsych really help? (2/2)

• Trivial to work on analysis before collecting human data

• Much easier to write up a good analysis plan, share it, improve it, …

• Create fully reproducible papers and results’ reports easily

• Sharing protocol, materials, data preparation is trivial (single command)

• Creating future-proof fully reproducible data preparation projects (with Docker) is one command away

jsPsychR ♥ RRs

More information

For more information about RRs, templates, checklists, list of participating journals (>300), etc.:

• https://www.cos.io/initiatives/registered-reports

Also, check out the future:

• RRs v2: Peer community in

And our manual:

• https://gorkang.github.io/jsPsychRmanual/

Thanks!

Gorka Navarrete

gorkang@gmail.com

https://fosstodon.org/@gorkang